Sensation imparting device, robot control system, and robot control method

ABSTRACT

According to the present invention, a terminal device has: a transmission unit which transmits, to a robot, operator state information indicating the state of a user operating the robot; a receiving unit which receives, from the robot, robot state information indicating the state of the robot; a sensation imparting unit which imparts a predetermined sensation to the user; and a control unit which, when a delay time required for the receiving unit to receive the robot state information after the transmission unit transmits the operator state information is no longer than a predetermined time, controls the sensation imparting unit so as to impart a sensation based on the robot state information to the user, and when the delay time is longer than the predetermined time, controls the sensation imparting unit to impart, to the user, a sensation based on virtual state information that indicates an estimated robot state.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is a continuation application of InternationalApplication No. PCT/JP2018/018883, filed on May 16, 2018, which claimspriority to Japanese Patent Application No. 2017-098532, filed on May17, 2017. The contents of these applications are incorporated herein byreference in their entirety.

BACKGROUND OF THE INVENTION

The present invention relates to a sensation imparting apparatus, arobot control system, and a robot control method.

Telexistence is known as a subset of virtual reality. Telexistence isalso said to be a technique that allows a person to have a sense ofremote reality or remote presence. Telexistence provides an environmentin which a user takes an action, through an avator robot, to an objector person at a remote location in real time while making the user feelas if the object or person is nearby.

In order to operate a robot in real time through a network bytelexistence, a high-speed network is required, and the network isrequired to have a very low latency and not to lose data. However,depending on the network in which data is transmitted, transmissiondelay or loss of data may occur.

In a case where data to be transferred is one piece of data transmittedin one direction, an influence of the transmission delay and the loss ofdata can be reduced by temporarily storing the data and stream in abuffer using a streaming algorithm that dynamically controls the rate.Also, by using streaming protocols such as Web Real-Time Communication(WebRTC) or HTTP Live Streaming (HLS), high-quality audio and videomedia can be transmitted synchronously in one direction.

The technology concerning remote control is disclosed in, for example,Japanese Examined Patent Application Publication No. S62-29196 andCharith Lasantha Fernando, Masahiro Furukawa, Tadatoshi Kurogi, ShoKamuro, Katsunari Sato, Kouta Minamizawa and Susumu Tachi, “Design ofTELESAR V for Transferring Bodily Consciousness in Telexistence”, 2012IEEE/RSJ International Conference on Intelligent Robots and Systems(Portugal), Oct. 7-12, 2012.

In telexistence, it is necessary to transmit or receive multiple mediastreams bi-directionally and synchronously among a plurality ofendpoints on the Internet in order to enable a user to operate a robotas if he/she were in the robot. The plurality of pieces of media may be,for example, motion data, audio data, video data, or tactile data.

In the related art, in a case where a delay occurs between an operationof a robot operator and an action of a robot in telexistence, thecontrol of the robot may become difficult. For example, if the delay islarge, the situation assumed by the operator does not coincide with thesituation visually recognized by the operator through an image capturingdevice provided in the robot, which causes dizziness in the operator,making it difficult for the operator to continue controlling the robot.

The delay is, for example, a delay generated in data communication vianetworks, a delay generated by performing data encoding and decodingprocessing, a delay generated by performing Analog to Digital (AID)transform processing on data (for example, a delay due to sampling), amechanical delay, or the like.

BRIEF SUMMARY OF THE INVENTION

This invention focuses on these points, and an object of the presentinvention is to provide a sensation imparting apparatus, a robot controlsystem, and a robot control method capable of improving operability intelexistence.

A sensation imparting apparatus according to the first aspect of thepresent invention comprises a transmission part that transmits operatorstate information indicating a state of an operator operating a robot tothe robot, a reception part that receives robot state informationindicating a state of the robot from the robot, a sensation impartingpart that imparts a predetermined sensation to the operator, and acontrol part that controls the sensation imparting part to impart asensation based on the robot state information to the operator if adelay time required from when the transmission part transmits theoperator state information to when the reception part receives the robotstate information is equal to or shorter than a predetermined timeperiod, and controls the sensation imparting part to impart a sensationbased on virtual state information indicating an estimated state of therobot to the operator if the delay time is longer than the predeterminedtime period.

A robot control system according to the second aspect of the presentinvention comprises a sensation imparting apparatus that transmits, to anetwork, operator state information indicating a state of an operatoroperating a robot, a robot control apparatus that controls the robot onthe basis of the operator state information received via the network,and a management apparatus that is capable of communicating with therobot control apparatus and the sensation imparting apparatus via thenetwork. The sensation imparting apparatus includes a transmission partthat transmits the operator state information indicating the state ofthe operator operating the robot to the robot, a reception part thatreceives robot state information indicating a state of the robot fromthe robot, a sensation imparting part that imparts a predeterminedsensation to the operator, and a control part that controls thesensation imparting part to impart a sensation based on the robot stateinformation to the operator if a delay time required from when thetransmission part transmits the operator state information to when thereception part receives the robot state information is equal to orshorter than a predetermined time period, and controls the sensationimparting part to impart a sensation based on virtual state informationindicating an estimated state of the robot to the operator if the delaytime is longer than the predetermined time period. The managementapparatus includes a delay specification part that specifies the delaytime, and a notification part that notifies the control part about thedelay time specified by the delay specification part.

A robot control method according to the third aspect of the presentinvention, the method comprises the computer-implemented steps oftransmitting operator state information indicating a state of anoperator operating a robot to the robot, determining whether or not adelay time required from transmitting the operator state information toreceiving robot state information indicating a state of the robot isequal to or less than a predetermined time period, and imparting asensation based on the robot state information to the operator if thedelay time is equal to or less than the predetermined time period, andimparting a sensation based on virtual state information indicating anestimated state of the robot to the operator if the delay time is longerthan the predetermined time period.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating a schematic configuration of arobot system according to an embodiment of the present invention.

FIG. 2 is a block diagram illustrating a schematic configuration of aterminal apparatus according to the embodiment of the present invention.

FIG. 3 is a block diagram illustrating a schematic configuration of arobot control apparatus according to the embodiment of the presentinvention.

FIG. 4 is a diagram for explaining how to adjust delay time by buffersin a terminal apparatus and a robot control apparatus according to theembodiment of the present invention.

FIG. 5 is a block diagram illustrating an image of a virtual robot inthe robot system according to the embodiment of the present invention.

FIG. 6A is a diagram for explaining the operation of the robot system ifthe delay time is equal to or less than the predetermined time period.

FIG. 6B is a diagram for explaining the operation of the robot system ifthe delay time is longer than the predetermined time period.

FIG. 7 is a block diagram illustrating a schematic configuration ofcommunication frames according to the embodiment of the presentinvention.

FIG. 8 is a block diagram illustrating a schematic configuration of atime-sharing communication frame according to the embodiment of thepresent invention.

FIG. 9 is a diagram illustrating a management apparatus capable ofcommunicating with a terminal apparatus and a robot control apparatusvia a network.

FIG. 10 is a diagram illustrating the robot system in which the robotcontrol apparatus provides the virtual state information to the user.

DETAILED DESCRIPTION OF THE INVENTION

Hereinafter, the present invention will be described through exemplaryembodiments of the present invention, but the following exemplaryembodiments do not limit the invention according to the claims, and notall of the combinations of features described in the exemplaryembodiments are necessarily essential to the solution means of theinvention.

[Outline of a Robot System]

FIG. 1 is a block diagram illustrating a schematic configuration of arobot system 1 according to an embodiment of the present invention. Therobot system 1 includes n (n is an integer of 2 or more) terminals 21-1to 21-n, n robots 31-1 to 31-n, n robot control apparatuses 41-1 to41-n, and a network 51. The network 51 is, for example, the Internet, ormay be any other network. The network 51 may be, for example, a wirednetwork, a wireless network, or a network including both wired andwireless networks.

FIG. 1 shows n users 11-1 to 11-n and n robots 31-1 to 31-n (n is aninteger of 1 or more). The robots 31-1 to 31-n include, for example, atotally humanoid robot or a partially humanoid robot. A partiallyhumanoid robot is a robot in which a part of the entire body is humanoidand another part is not humanoid, the part being, for example, an upperbody part, a lower body part, or an arm part. Each of the robots 31-1 to31-n may be any other form of robot. The robots 31-1 to 31-n may have,for example, a robot operating system (ROS) function.

Here, in the example of FIG. 1, the respective users 11-1 to 11-n andthe respective terminal apparatuses 21-1 to 21-n are associated witheach other. Further, in the example of FIG. 1, the respective robots31-1 to 31-n and the respective robot control apparatuses 41-1 to 41-nare associated with each other. Furthermore, in the example of FIG. 1,the respective terminal apparatuses 21-1 to 21-n and the respectiverobot control apparatuses 41-1 to 41-n are associated with each other.Thus, in the example of FIG. 1, one user 11-i (i is any integer of1≤i≤n), one terminal apparatus 21-i, one robot 31-i, and one robotcontrol apparatus 414 are associated with each other.

In the embodiment, then terminal apparatuses 21-1 to 21-n have the samefunction, the n robot control apparatuses 41-1 to 41-n have the samefunction, and the n robots 31-1 to 31-n have the same function. Also,the n users 11-1 to 11-n are different persons, but have the samefunction in that they operate the corresponding robots 31-1 to 31-n.Therefore, a combination of a user 11-1, a terminal apparatus 21-1, arobot 31-1, and a robot control apparatus 41-1 will be described as anexample below.

[Outline of a Terminal Apparatus]

FIG. 2 is a block diagram illustrating a schematic configuration of theterminal apparatus 21-1 according to the embodiment of the presentinvention. The terminal apparatus 21-1 detects a state of the user 11-1and also functions as a sensation imparting apparatus for impartingvarious types of sensations to the user 11-1. The sensation imparted bythe terminal apparatus 21-1 to the user 11-1 is arbitrary, but is, forexample, a visual, auditory, tactile, or force sensation. The terminalapparatus 21-1 imparts a sensation based on a state of the robot 31-1acquired via the robot control apparatus 41-1 to the user 11-1.

The terminal apparatus 21-1 includes a detection part Iii, a buffer 112,a communication part 113, a sensation imparting part 114, and a controlpart 115. The communication part 113 includes a transmission part 131and a reception part 132. The control part 115 includes a storage part151 including a read only memory (ROM) and a random access memory (RAM).

The detection part 111 detects the state of the user The state detectedby the detection part 111 is, for example, a motion of fingers of theuser 11-1, a motion of a torso of the user 11-1, or a motion of eyes(for example, a line of sight) of the user 11-1.

The detection part 111 includes, for example, a sensor for detecting astate of a detection target. The detection part 111 includes, forexample, a sensor for a head, a sensor for fingers, and a sensor for atorso. The sensor for the head is provided in, for example, ahead-mounted display that provides a video image and sound to the user11-1. The head-mounted display includes, for example, a sensor fordetecting the motion of the user's eyes (for example, a line of sight).The sensor may detect the motion of the left eye and the motion of theright eye separately.

The sensor may be, for example, a sensor in the form of a sensorattached to the user 11-1, or may be a sensor not attached to the user11-1. The sensor not attached to the user 11-1 is, for example, an imagecapturing device (for example, a camera) that captures an image forexample, a video image) of all or a part of the user 11-1.

The buffer 112 is a memory for temporarily storing data. In theembodiment, the buffer 112 has a function of temporarily storing data tobe transmitted by the transmission part 131 and a function oftemporarily storing data received by the reception part 132. A differentbuffer may be provided for each of these two functions. The buffer 112temporarily stores one or more pieces of robot state informationreceived by the reception part 132, and then sequentially outputs theone or more pieces of robot state information to the sensation impartingpart 114. The robot state information is information indicating thestate of the robot 31-1, and is information, for example, indicating animage captured by the robot 31-1, a pressure detected by the robot 31-1,or a temperature detected by the robot 31-1.

The buffer 112 temporarily stores operator state information indicatinga state of an operator detected by the detection part 111, and thensequentially outputs one or more pieces of the operator stateinformation to the transmission part 131 The operator state informationis, for example, information indicating the position of the body of theuser 11-1 detected by using a sensor attached by the user 11-1 tohis/her body.

The communication part 113 communicates with the robot control apparatus41-1 corresponding to the terminal apparatus 21-1. The communicationpart 113 communicates with the robot control apparatus 41-1 via thenetwork 51. Also, the communication part 113 communicates via WebRTC.The communication part 113 can communicate at a high frame rate and witha low delay, for example, via WebRTC.

The transmission part 131 acquires the operator state informationindicating the state of the user 11-1 operating the robot 31-1 from thebuffer 112, and transmits the acquired operator state information to therobot 31-1. Specifically, the transmission part 131 transmits data to betransmitted including the operator state information to the robotcontrol apparatus 41-1.

The reception part 132 receives the robot state information indicatingthe state of the robot 31-1 as data transmitted from the robot controlapparatus 41-1. The reception part 132 inputs the received robot stateinformation to the buffer 112.

The sensation imparting part 114 imparts a sensation based on the robotstate information received by the reception part 132 to the user 11-1.That is, the sensation imparting part 114 imparts the sensationcorresponding to the robot state information indicating the state of therobot 31-1 received by the reception part 132 to the user 11-1.

The sensation imparting part 114 includes, for example, a device forimparting a sensation to the user 11-1. For example, the sensationimparting part 114 may include a video display device for displaying animage (video) to impart a visual sensation to the user 11-1, a speakerfor outputting sound (audio) to impart an auditory sensation to the user11-1, and a motion generator for generating a pressure or a vibration(haptics) to impart a tactile sensation to the user 11-1. The videodisplay device and the speaker are provided, for example, in a headdevice (such as a head-mounted display). The device for imparting thesensation to the user 11-1 may be a device attached to the user 11-i ora device not attached to the user 11-1. All or a part of the detectionpart 111 and the sensation imparting part 114 may be included in thesame device.

The control part 115 performs various controls in the terminal apparatus21-1. The control part 115 includes, for example, a central processingunit (CPU) and a storage part 151 for storing data. The control part 115performs the various controls by executing programs stored in thestorage part 151 using, for example, parameters stored in the storagepart 151.

In the embodiment, the control part 115 can adjust the length of timefor temporarily storing the data to be transmitted with the buffer 112,and can adjust the length of time for temporarily storing the receiveddata with the buffer 112.

Here, the terminal apparatus 21-1 may be composed of a plurality ofdevices, and may include, for example, the detection part 111, thebuffer 112, the communication part 113, the sensation imparting part114, and the control part 115 shown in FIG. 2 in a distributed manner intwo or more devices. Further, the transmission part 131 and thereception part 132 may be provided in different devices. For example,the terminal apparatus 211 may be configured by using a device includingthe detection part 111, a device including the sensation imparting part114, and a device including the buffer 112, the communication part 113,and the control part 115. In addition, the device including thedetection part 111 and the device including the sensation imparting part114 may be partially or entirely integrated with each other.

[Outline of a Robot Control Apparatus]

FIG. 3 is a block diagram illustrating a schematic configuration of therobot control apparatus 41-1 according to the embodiment of the presentinvention. The robot control apparatus 41-1 includes a communicationpart 211, a buffer 212, a robot driving part 213, a detection part 214,and a control part 215. The communication part 211 includes a receptionpart 231 and a transmission part 232. The control part 215 includes aCPU and a storage part 251 including a ROM and a RAM.

The communication part 211 communicates with the terminal apparatus 21-1corresponding to the robot control apparatus 41-1. In the embodiment,the communication part 211 communicates via the network 51. Thecommunication part 211 communicates via WebRTC, for example. Thereception part 231 receives data such as the operator state informationtransmitted from the terminal apparatus 21-1. The transmission part 232transmits data such as the robot state information to be transmitted tothe terminal apparatus 21-1.

The buffer 212 is a memory for temporarily storing data. In theembodiment, the buffer 212 has a function of temporarily storing therobot state information to be transmitted by the transmission part 232,and a function of temporarily storing the operator state informationreceived by the reception part 231. It should be noted that the buffer212 may have different buffers for each of these two functions.

The robot driving part 213 drives a driving part of the robot 31-1corresponding to the robot control apparatus 41-1 on the basis of theoperator state information received from the terminal apparatus 21-1.Here, the driving part of the robot 31-1 is, for example, a driving partfor driving a head, a driving part for driving fingers, or a drivingpart for driving a torso.

The detection part 214 detects motion of the robot 31-1. The motiondetected by the detection part 214 is, for example, the motion of thefingers of the robot 31-1, the motion of the torso of the robot 31-1, orthe motion of the eyes (e.g., a line of sight) of the robot 31-1.

The detection part 214 includes, for example, the sensor for detectingthe state of the detection target. As an example, the detection part 214includes the sensor for the head, the sensor for the fingers, and thesensor for the torso. The sensor for the head is, for example, an imagecapturing device (a camera) for capturing a video image or a microphonefor inputting a sound. For example, the detection part 214 mayseparately process the video image corresponding to the left eye and thevideo image corresponding to the right eye.

The sensor may be a sensor attached to the robot 31-1 or a sensor notattached to the robot 31-1. The sensor not attached to the robot 31-1is, for example, an image capturing device which is installed in thevicinity of the robot 31-1 and captures an image (for example, a videoimage). Various types of data are captured by sampling using, forexample, A/D conversion. In the embodiment, the data detected by thedetection part 214 is the robot state information to be transmitted. Itshould be noted that all or a part of the robot driving part 213 and thedetection part 214 may be, for example, included in the same device.

The control part 215 performs various controls in the robot controlapparatus 41-i. The control part 215 includes, for example, a CPU and astorage part 251 for storing data. The control part 215 performs thevarious controls by, for example, executing programs stored in thestorage part 251 using parameters stored in the storage part 251. In theembodiment, the control part 215 can adjust the length of time fortemporarily storing data such as the robot state information to betransmitted in the butler 212, and can adjust the length of time fortemporarily storing data such as the received operator state informationin the buffer 212.

Here, the robot control apparatus 41-1 may be composed of a plurality ofdevices, and may include, for example, the communication part 211, thebuffer 212, the robot driving part 213, the detection part 214, and thecontrol part 215 shown in FIG. 3 in a distributed manner in two or moredevices. Further, the reception part 231 and the transmission part 232may be provided in different devices. For example, the robot controlapparatus 41-1 may be configured using a device including the robotdriving part 213, a device including the detection part 214, and adevice including the communication part 211, the buffer 212, and thecontrol part 215. In addition, the device including the robot drivingpart 213 and the device including the detection part 214 may bepartially or entirely integrated with each other.

[Outline of an Operation in the Terminal Apparatus and the Robot ControlApparatus]

The terminal apparatus 21-1 detects the state of the user 11-1 with thedetection part 111, and transmits the operator state informationincluding the detected data to the robot control apparatus 41-1 via thenetwork 51. The robot control apparatus 41-1 receives the operator stateinformation transmitted from the terminal apparatus 21-1 via the network51, and drives the robot 31-1 using the robot driving part 213 on thebasis of the received operator state information. The robot 31-1 isdriven and operated by the robot (hiving part 213 of the robot controlapparatus 41-1.

The robot control apparatus 41-1 detects the state of the robot 31-1 bythe detection part 214, and transmits the robot state informationincluding the detected data to the terminal apparatus 21-1 via thenetwork 51. The terminal apparatus 21-1 receives the robot stateinformation transmitted from the robot control apparatus 41-1 via thenetwork 51, and imparts the sensation to the user 11-1 with thesensation imparting part 114 on the basis of the received robot stateinformation.

Here, in the embodiment, the terminal apparatus 21-1 and the robotcontrol apparatus 41-1 are associated with each other before being usedby the user 11-1 or when being used by the user 11-1. The terminalapparatus 21-1 stores information of the address of the robot controlapparatus 41-1 corresponding to its own apparatus (the terminalapparatus 21-1), and the robot control apparatus 41-1 stores informationof the address of the terminal apparatus 21-1 corresponding to its ownapparatus (the robot control apparatus 41-1). Then, the terminalapparatus 21-1 and the robot control apparatus 41-1, which correspond toeach other, communicate with each other as communication partners byusing the stored address information, for example, by including theaddress information in a signal to be transmitted.

[The Terminal Apparatus and the Robot Control Apparatus: Adjusting DelayTime Using the Buffers]

FIG. 4 is a diagram for explaining how to adjust delay time usingbuffers in the terminal apparatus 21-1 and the robot control apparatus41-1 according to the embodiment of the present invention. FIG. 4 showsthe user 11-1, a head device 311, finger devices 312-1 to 312-2, andtorso devices 313-1 to 313-2, which are constituent parts of theterminal apparatus 21-1, as well as the buffer 112, which is aconstituent part of the terminal apparatus 21-1. Further, FIG. 4 showsthe robot 31-1, a visual device 411, audio devices 412-1 to 412-2, andfinger devices 413-i to 413-2, which are constituent parts of the robotcontrol apparatus 41-1, as well as the buffer 212, which is aconstituent part of the robot control apparatus 41-1.

The user 11-1 wears the head device 311 on the head, the finger device312-1 on the fingers of his/her right hand, the finger device 312-2 onthe fingers of his/her left hand, the torso device 313-1 on the right ofhis/her torso, and the torso device 313-2 on the left of his/her torso.The head device 311 includes a visual part 331 at a positioncorresponding to both eyes of the user 11-1, an audio part 332-1 at aposition corresponding to the right ear of the user 11-1, and an audiopart 332-2 at a position corresponding to the left ear of the user 11-1.The head device 311 is, for example, a head-mounted display. The visualpart 331 processes data for each of the tight eye and the left eye.

Here, the head device 311, the finger devices 312-1 to 312-2, and thetorso devices 313-1 to 313-2 have the function of the detection part 111of the terminal apparatus 21-1. The head device 311 detects the motionof the head. The finger devices 312-1 to 312-2 detect the motion of thefingers of the right and left hands, respectively. The torso devices313-1 to 313-2 detect the motion of the torso. The terminal apparatus21-1 realizes a motion capture function for detecting the motion (changein posture) of the user 11-1 on the basis of the motion of each partdetected by the head device 311, the finger devices 312-1 to 312-2, andthe torso devices 313-1 to 313-2.

In the example of FIG. 4, the user 11-1 has the right and left torsodevices 313-1 to 313-2 separately, but the user 11-1 may have one torsodevice instead of the right and left torso devices 313-1 to 313-2. Itshould be noted that FIG. 4 is an example, and any other device fordetecting the state of the user may be used as the detection part 111.For example, a touch panel for detecting the content of an operationusing fingers or the like of a person may be used.

The finger devices 312-1 to 312-2, the torso devices 313-1 to 313-2, thevisual part 331, and the audio parts 332-1 to 332-2 have the function ofthe sensation imparting part 114 of the terminal apparatus 21-1. Thefinger devices 312-1 to 312-2 each have a function of imparting thetactile sensation to each of the fingers. The torso devices 313-1 to313-2 have a function of imparting the tactile sensation to the torso.The visual part 331 has the function of imparting the visual sensationto both eyes (each of the left and right eyes) of the user 11-1, anddisplays, for example, a video image. The audio parts 332-1 to 332-2have the function of imparting the auditory sensation to both ears (eachof the left and right ears) of the user 11-1, and output sound, forexample. FIG. 4 is an example, and the sensation imparting part 114 maybe any device that imparts other sensations to the user.

The robot 31-1 is equipped with a visual device 411 in a positioncorresponding to both eyes of a person, an audio device 412-1 in aposition corresponding to a right ear of a person, an audio device 412-2in a position corresponding to a left ear of a person, a finger device413-1 in a position corresponding to fingers of a right hand of aperson, and a finger device 413-2 in a position corresponding to fingersof a left hand of a person.

Here, the visual device 411, the audio devices 412-1 to 412-2, and thefinger devices 413-1 to 413-2 have the function of the detection part214 of the robot control apparatus 41-1. The visual device 411 includes,for example, an image capturing device, and generates video data as datarelating to the visual sensation. In the embodiment, the visual device411 processes the video data for each of the right eye and the left eye.The audio devices 412-1 to 412-2 generate audio data as data relating tothe auditory sensation.

The finger devices 413-1 to 413-2 generate data relating to the tactilesensation for the respective fingers. The tactile sensation data is dataindicating the degree of hardness and roughness of a place touched bythe finger devices 413-1 to 413-2. The detection part 214 may include adevice detachable from the robot 31-1. Further, as another configurationexample, the detection part 214 may include a device that is integratedand fixed to the robot 31-1. In this case, the device may be regarded asa part of the robot 31-1.

It should be noted that illustration of the communication part 113 andthe control part 115 of the terminal apparatus 21-1 is omitted in theexample shown in FIG. 4. Further, illustration of the communication part211, the robot driving part 213, and the control part 215 of the robotcontrol apparatus 41-1 is omitted in the example shown in FIG. 4.Furthermore, for convenience of description, the buffer 112 and thebuffer 212 are shown as a single unit in the example shown in FIG. 4,but the configuration is not necessarily limited to this, and the buffer112 is provided in the terminal apparatus 21-1 while the buffer 212 isprovided in the robot control apparatus 41-1.

In the example shown in FIG. 4, the operator state informationindicating the state of the user 11-1 detected by the detection part 111of the terminal apparatus 21-1 is transmitted to the control part 215 orthe robot driving part 213 of the robot control apparatus 41-1 by atransmission session P1, a transmission session P2, and a transmissionsession P3. The transmission session P1 is a session from the controlpart 115 or the detection part 111 of the terminal apparatus 21-1 to thebuffer 112 of the terminal apparatus 21-1. The transmission session P2is a session from the buffer 112 of the terminal apparatus 21-1 to thebuffer 212 of the robot control apparatus 41-1. The transmission sessionP3 is a session from the buffer 212 of the robot control apparatus 41-1to the control part 215 or the robot driving part 213 of the robotcontrol apparatus 41-1.

The robot state information indicating the states of the robots 31-1 to31-n detected by the detection part 214 of the robot control apparatus41-1 is transmitted to the control part 115 or the sensation impartingpart 114 of the terminal apparatus 21-1 by a transmission session Pit, atransmission session P12, and a transmission session P13. Thetransmission session P11 is a session to the buffer 212 of the robotcontrol apparatus 41-1. The transmission session P12 is a session fromthe buffer 212 of the robot control apparatus 41-1 to the buffer 112 ofthe terminal apparatus 21-1. The transmission session P13 is a sessionfrom the buffer 112 of the terminal apparatus 21-1 to the control part115 or the sensation imparting part 114 of the terminal apparatus 21-1.

The data transmission between the buffer 112 of the terminal apparatus21-1 and the buffer 212 of the robot control apparatus 41-1 is performedby the communication part 113 of the terminal apparatus 21-1 and thecommunication part 211 of the robot control apparatus 41-1 via thenetwork 51.

As shown in FIG. 4, in the embodiment, there are mainly two sessionsserving as bi-directional data communication between the terminalapparatus 21-1 and the robot control apparatus 41-1. The first sessionis a session including the transmission sessions P1 to P3 from theterminal apparatus 21-1 to the robot control apparatus 41-1. In thefirst session, for example, the operator state information indicatingthe state of the user 11-1 is transmitted. The operator stateinformation is, for example, information including data indicating themotion of the user's eyes, the motion of the user's fingers, or themotion of the user's torso.

The second session is a session including the transmission sessions P11to P13 from the robot control apparatus 41-1 to the terminal apparatus21-1. In the second session, the robot state information including, forexample, the video data, the audio data, and the tactile data indicatingthe state of the robot 31-1 is transmitted.

The robot 31-1 operates in accordance with the motion of the user 11-1.The user 11-1 sees an image, hears a sound, and feels a tactilesensation at the position of the robot 31-1, and performs an operation.The robot 31-1 and the user 11-1 repeat these operations.

The robot system 1 of the embodiment compensates for the delay ofbi-directional communication in the first session and the secondsession. For example, the control part 115 controls the amount of therobot state information temporarily stored in the buffer 112 so that thevariation amount in the delay time required from the transmission of theuser 11-1 state information by the transmission part 131 to thereception of the robot state information by the reception part 132 fallswithin a predetermined range. Here, for example, the user 11-1 and therobot 31-1 may have different delay times for their respective sessions.In such a case, the robot system 1 compensates for a different amount ofthe delay time for each of the first session and the second session inorder to achieve synchronized motion on the side of the user 11-1 and onthe side of the robot 31-1.

In this example, the total delay time of the transmission sessions P1 toP3 from the user 11-1 side (the terminal apparatus 21-1) to the robot31-1 side (the robot control apparatus 41-1) is represented by LatM.Further, the total delay time of the transmission sessions P11 to P13from the robot 31-1 side (the robot control apparatus 41-1) to the user11-1 side (the terminal apparatus 21-1) is represented by LatN. Inaddition, the delay time of the transmission session P1 is representedby LatMu, the delay time of the transmission session P3 is representedby LatMr, the delay time of the transmission session P11 is representedby LatNr, and the delay time of the transmission session P13 isrepresented by LatNu.

The delay time in a mechanism (in the embodiment, the terminal apparatus21-1) for detecting the motion of the user 11-1 is represented by LatMm,and the delay time in the mechanism (in the present embodiment, therobot control apparatus 41-1) for driving the robot 31-1 is representedby LatNm. In this case, the equations (1) and (2) are established.

[Equation 1]

LatM=LatMu+LatMr+LatMm   (1)

[Equation 2]

LatN=LatNr+LatNu+LatNm   (2)

The equation (3) is established when a total round-trip delay time (thedelay time of a round-trip) including the transmission sessions P1 to P3and the transmission sessions P11 to P13 is expressed by RTT in a casewhere the side of the robot 31-1 is seen from the side of the user 11-1(for example, the side of the control part 115 of the terminal apparatus21-1).

[Equation 3]

RTT=LatM+LatN   (3)

If RTT becomes large, the user 11-1 may feel dizziness and feel ill.Also, if the variation amount of RTT over time is large, the user 11-1may feel ill.

Therefore, the robot system 1 of the embodiment is configured such thatthe buffer 112 is provided on the side of the user 11-1 that is on oneside of the network 51, and the buffer 212 is provided on the side ofthe robot 31-1 that is on the other side of the network 51, so that thedelay time of the data in each of the buffer 112 and the butter 212 canbe adjusted. That is, the robot system can adjust the delay time on eachof the side of the user 11-1 and the side of the robot 31-1.

On the side of the user 11-1, the control part 115 of the terminalapparatus 21-1 controls the delay time of the data in the buffer 112 inreal time so that the equation (4) is established. On the side of therobot 31-1, the control part 215 of the robot control apparatus 41-1controls the delay time of the data in the buffer 212 in real-time sothat the equation (5) is established.

[Equation 4]

LatMu˜LatNu   (4)

[Equation 5]

LatMr˜LatNr   (5)

Here, in the equations (4) and (5), “˜” represents the same or close.That is, the control part 115 of the terminal apparatus 21-1 controlsthe delay time so that the difference between the delay time LatMu ofthe transmission session P1 and the delay time LatNu of the transmissionsession P13 is equal to or less than a predetermined first time periodso that the equation (4) is established. Also, the control part 215 ofthe robot control apparatus 41-1 controls the delay time so that thedifference between the delay time LatMr of the transmission session P3and the delay time LatNr of the transmission session P11 is equal to orless than a predetermined second time period so that the equation (5) isestablished. The predetermined first time period and the predeterminedsecond time period are, for example, the maximum values of a time periodthat does not cause a hindrance in the operation of the robot 31-1 bythe user 11-1.

Here, in this example, it is assumed that the video data transmittedfrom the robot control apparatus 41-1 to the terminal apparatus 21-1requires a longer transmission time than that of the data of other media(such as the audio data and the tactile data). Regardless of the type ofmedia, the control part 115 and the control part 215 control the delaytimes in the buffer 112 and the buffer 212 so that the difference in theround-trip delay time (in this example, RTT) between the side of theuser 11-1 and the side of the robot 31-1 falls within the predeterminedrange (within the predetermined time period). The predetermined timeperiod is determined on the basis of the video data requiring thelongest transmission time, and is, for example, 40 ms. The control part115 and the control part 215 may allow the data of media whosedifference between the delay times is within the predetermined timeperiod to pass through the buffer 112 and the buffer 212 without addingthe delay time.

[Imparting a Virtual Sensation]

Even if the operability is improved by suppressing the variation of theround-trip delay time between the side of the user 11-1 and the side ofthe robot 31-1, the user 11-1 may feel uncomfortable if the delay timeis long. In particular, if there is a long delay time from the operationof the user 11-1 until the user 11-1 visually recognizes the imagecaptured by the robot 31-1 in the posture after being changed inaccordance with the operation of the user 11-1, the user 11-1 is likelyto feel dizziness. Therefore, the terminal apparatus 21 is configured tobe able to impart a virtual sensation to the user 11-1. Hereinafter, anembodiment will be exemplified in which the terminal apparatus 21 causesthe user 11-1 to visually recognize a virtual visual image generated byestimating an image visually recognized by the robot 31-1 as a virtualsensation. The virtual sensation is not limited to the virtual visualsensation, and may be other sensations such as a virtual auditorysensation, a virtual tactile sensation, or a virtual force sensation.

FIG. 5 is a block diagram showing an image of a configuration forproviding the virtual sensation in the robot system 1 according to theembodiment of the present invention. FIG. 5 shows the user 11-1 wearingthe head device 311 or the like, and a virtual robot 711. The virtualrobot 711 is a robot capable of providing the virtual sensation usingdata stored in the buffer 112 of the terminal apparatus 21-1, andincludes a computer that virtually reproduces the state of the robot31-1. In this example, the control part 115 of the terminal apparatus21-1 operates as if it is transmitting/receiving data to/from thevirtual robot 711.

If the delay time (for example, RTT) required from the transmission ofthe operator state information by the transmission part 131 to thereception of the robot state information by the reception part 132 isequal to or less than the predetermined time period, the control part115 controls the sensation imparting part 114 to impart the sensationbased on the robot state information to the user 11-1 as an operator. Onthe other hand, if the delay time is longer than the predetermined timeperiod, the control part 115 controls the sensation imparting part 114to impart the sensation based on virtual state information indicatingthe estimated state of the robot 31-1 to the user 11-1. The virtualstate information includes data indicating visual, auditory, tactile,and force sensations, in the same manner as the robot state information.

FIG. 6A is a diagram for explaining the operation of the robot system 1if the delay time is equal to or less than the predetermined timeperiod. FIG. 6B is a diagram for explaining the operation of the robotsystem 1 if the delay time is longer than the predetermined time period.As shown in FIG. 6A, if the delay time is small enough not to causetrouble in the operation of the user 11-1, the robot state informationis imparted to the user 11-1 via the buffer 112 of the terminalapparatus 21-1. On the other hand, as shown in FIG: 6B, if the delaytime is long enough to cause trouble in the operation of the user 11-1,for example, the virtual state information stored in the storage part151 is imparted to the user 11-1 from the terminal apparatus 21-1.Hereinafter, the details of the operation of the terminal apparatus 21-1for providing the virtual state information will be described.

The control part 115 controls the sensation imparting part 114 so as toimpart, to the user 11-1, at least some pieces of the virtual stateinformation among a plurality of pieces of the virtual state informationcorresponding to virtual robots in different states, which are stored inthe storage part 151, for example. Specifically, the control part 115controls the sensation imparting part 114 to impart, to the user 11-1,the sensation based on the virtual state information selected from aplurality of pieces of the virtual state information on the basis of theoperator state information.

More specifically, the storage part 151 stores the plurality of piecesof the virtual state information including virtual images correspondingto images captured by the robots 31-1 in different states. The virtualimage is a captured image generated by the visual device 411 alreadyincluded in the robot 31-1 or an image capturing device mounted onanother equivalent robot, and the image is associated with informationindicating the state of the robot (for example, the position of eachpart of the robot) at the time of capturing the image. The control part115 controls the sensation imparting part 114 so as to impart, to theoperator, the sensation based on the virtual state information, amongthe plurality of pieces of the virtual state information, including thevirtual images corresponding to the position of the robot 31-1 specifiedon the basis of operation content indicated by the operator stateinformation.

In order to select one piece of the virtual state information from theplurality of pieces of the virtual state information, the control part115 estimates the state of the robot 31-1 after the operation of theuser 11-1 on the basis of the state of the user 11-1 indicated by theimmediately preceding operator state information and the state of theuser 11-1 indicated by the most recent operator state information. Thecontrol part 115 selects one piece of the virtual state informationcorresponding to the estimated state of the robot 31-1. In this manner,even if the robot state information transmitted from the robot controlapparatus 41-1 is delayed, the control part 115 can impart the sensationbased on the virtual state information to the operator before thereception part 132 receives the robot state information. This allows theuser 11-1 to easily operate the robot 31-1.

The control part 115 may extract an image of the robot 31-1 included inan image as the robot state information that was provided to the user11-1 immediately before, and may provide the user 11-1 with an imageobtained by replacing the extracted image with an image of the robotindicated by the virtual state information. In this manner, the controlpart 115 can provide the user 11-1 with an image including an image of avirtual robot attempting to grasp an object when, for example, the robot31-1 is operating to grasp the object.

The control part 115 may store sensation history information indicatingcontent of the sensation based on the robot state information impartedby the sensation imparting part 114 in the storage part 151. In thisinstance, the control part 115 selects the virtual state information,from the plurality of pieces of the virtual state information, to beused for imparting the sensation to the operator on the basis of thecontent of the sensation indicated by the sensation history informationstored in the storage part 151 immediately before. For example, thecontrol part 115 selects the virtual state information in which thevariation amount from the sensation indicated by the sensation historyinformation stored immediately before falls within the predeterminedrange. By doing so, since the sensation imparted to the user 11-1 doesnot change abruptly, the terminal apparatus 211 can prevent the user11-1 from feeling discomfort.

After imparting the sensation based on the virtual state information tothe user 11-1, the control part 115 controls the sensation impartingpart 114 to impart the sensation based on the robot state informationreceived by the reception part 132 to the user 11-1. Specifically, forexample, if it is determined that the motion of the robot 31-1 withinthe delay time required from the transmission of the operator stateinformation by the transmission part 131 to the reception of the robotstate information by the reception part 132 is smaller than an amountperceivable by the user 11-1, the control part 115 imparts the sensationbased on the actual state of the robot 31-1 to the user 11-1. Morespecifically, the control part 115 stops the impartation of a firstsensation based on the virtual state information and switches to theimpartation of a second sensation based on the robot state informationwhen it is determined that the variation amount of the robot stateinformation within the delay time is less than a threshold value. Insuch a way, the user 11-1 can grasp the actual state of the robot 31-1at the point in time when the motion of the robot 31-1 stops.

Here, the state indicated by the virtual state information does notnecessarily coincide completely with the actual state of the robot 31-1.Therefore, after the impartation of the first sensation based on thevirtual state information is stopped and the impartation of the secondsensation based on the robot state information is started, the user 11-1may have a sense of discomfort because the difference between the firstsensation and the second sensation is large. To solve this problem, whenthere is a difference of a predetermined magnitude or more between thefirst sensation based on the virtual state information and the secondsensation based on the robot state information received by the receptionpart 132 after the sensation based on the virtual state information isimparted to the user 11-1, the control part 115 may control thesensation imparting part 114 so as to impart a sensation interpolatedbetween the first sensation and the second sensation to the user 11-1.

The control part 115, for example, generates the plurality of pieces ofthe virtual state information for interpolating between the firstsensation and the second sensation and controls the sensation impartingpart 114 so that the user gradually reaches the second sensation fromthe first sensation. In a case where the first sensation and the secondsensation are visual sensations, the control part 115 provides aplurality of virtual images to the user 11-1 so that the fingertip ofthe robot 31-1 at the position corresponding to the first sensationincluded in the image provided to the user 11-1 appears to movegradually to the position corresponding to the second sensation. Thisallows the control part 115 to prevent the user 11-1 from feelingdiscomfort due to a sudden change in the imparted sensation.

The control part 115 may control the sensation imparting part 114 toimpart the sensation based on the robot state information to the user11-1 if the amount of the one or more pieces of the robot stateinformation stored in the buffer 112 is equal to or greater than thethreshold value, and to impart the sensation based on the virtual stateinformation to the user 11-1 if the amount of the one or more pieces ofthe robot state information stored in the buffer 112 becomes less thanthe threshold value. By doing so, even if a transmission delay of therobot state information occurs in a state where the amount of the robotstate information temporarily stored in the buffer 112 is reduced, it ispossible to impart the sensation based on the virtual state informationto the user 11-1. Accordingly, the delay time required from thetransmission of the operator state information by the transmission part131 to the reception of the robot state information by the receptionpart 132 can be reduced, and the user 11-1 can easily control the robot31-1.

The control part 115 may specify a factor of the delay from thetransmission of the user 11-1 state information by the transmission part131 to the reception of the robot state information by the receptionpart 132. On the basis of the specified factor, the control part 115determines whether to control the sensation imparting part 114 to impartthe sensation based on the virtual state information stored in thestorage part 151 to the user 11-1, or to control the sensation impartingpart 114 to impart the sensation based on the virtual state informationreceived via the network 51 to the user 11-1. The virtual stateinformation received via the network 51 is, for example, the virtualstate information provided from the robot control apparatus 41-1 or aserver on a cloud that monitors the state of the robot 31-1 andgenerates the virtual state information on the basis of the state of therobot 31-1.

If a throughput or storage capacity of the robot control apparatus 41-1or the server on the cloud is better than that of the terminal 21-1, itis considered that the robot control apparatus 41-1 or the server on thecloud can generate the virtual state information with higher accuracy(that is, closer to the actual state of the robot 31-1) than the virtualstate information stored in the storage part 151. Therefore, the controlpart 115 can impart a sensation close to the actual state of the robot31-1 to the user 11-1 by using the virtual state information providedfrom the robot control apparatus 41-1 or the server on the cloud whenthe delay time in the network 51 is equal to or less than the thresholdvalue.

It should be noted that the control part 115 of the terminal apparatus21-1 and the control part 215 of the robot control apparatus 41-1communicate information about a processing delay time on their own sidewith each other via the network 51. The processing delay time is (i) atime from when the user 11-1 operates to when the terminal apparatus21-1 transmits the operator state information, and (ii) a time from whenthe robot 31-1 changes the state to when the robot control apparatus41-1 transmits the robot state information. The control part 115 and thecontrol part 215 specify a value of the round-trip delay time RTTbetween the side of the user 11-1 and the side of the robot 31-1 on thebasis of the acquired delay time of the other side. The control part 115determines whether to impart the sensation based on the virtual stateinformation to the user 11-1 or to impart the sensation based on therobot state information to the user 11-1 on the basis of the specifieddelay time value.

The robot system I may control the robot 31-1 on the basis of virtualoperator state information generated by estimating the state of the user11-1. For example, if the delay time from when the transmission part 131of the terminal apparatus 21-1 transmits the operator state informationto when the operator state information reaches the reception part 231 ofthe robot control apparatus 41-1 is equal to or longer than apredetermined threshold value, the control part 215 of the robot controlapparatus 41-1 controls the robot 31-1 on the basis of the virtualoperator state information. The virtual operator state information is,for example, information generated by estimating the most recent stateof the operator on the basis of one or more pieces of the immediatelypreceding operator state information. This makes it possible toaccelerate the response of the robot 31-1 to the operation performed bythe user 11-1, thereby further improving the operability.

The virtual operator state information used by the control part 215 maybe virtual operator state information selected from a plurality ofpieces of the virtual operator state information stored in the storagepart 251, or may be virtual operator state information generated on thebasis of a model of the user 11-1 generated by computer graphics (CG).

[Synchronizing Multi-Stream Data in the Network]

In the embodiment, the second session from the robot control apparatus41-1 to the terminal apparatus 21-1 includes the video data, audio data,and tactile data. Depending on the processing speed of the data ofrespective media, the delay times of the data of respective media may bedifferent. Among these media, the delay time of the video data is thelargest. Therefore, in the embodiment, the control part 115 of theterminal apparatus 21-1 and the control part 215 of the robot controlapparatus 41-1 use time information assigned to respective image framesincluded in the video data as a reference value of the delay time.

In one example, the control part 215 of the robot control apparatus 41-1includes the audio data and the tactile data in each of the image framesthat transmit the video data. This makes it possible to associate thetime information with the audio data and the tactile data to which thetime information is not assigned.

Alternatively, the control part 215 of the robot control apparatus 41-1may synchronize the video data, the audio data, and the tactile datausing time stamps. In this instance, the control part 215 of the robotcontrol apparatus 41-1 assigns a time stamp indicating the time to thedata or the communication frame of the respective media. The controlpart 115 of the terminal apparatus 21-1 and the control part 215 of therobot control apparatus 41-1 may use a clock signal supplied from ashared clock source as a clock used for referencing the time stamp. Asthe shared clock source, for example, a source provided by a third partymay be used, or a source provided by any device included in the robotsystem 1 (for example, a server that is a management apparatus connectedto the network 51) may be used.

FIG. 7 is a block diagram illustrating a schematic configuration ofcommunication frames F1 to Fj (j is an integer of 2 or more) accordingto the embodiment of the present invention. FIG. 7 shows a plurality (jpieces) of the consecutive communication frames F1 to Fj transmittedfrom the robot control apparatus 41-1.

In the embodiment, it is assumed that the communication speed of thecommunication frames F1 to Fj is 60 [fps (frames/second)]. Assuming thatthe transmission time T1 of the communication frame F1 is 0 [ms], thetransmission time Tj of the communication frame Fj is approximately{16.6666*(j−1)}[ms]. Each of the communication frames F1 to Fj includesthe video data, imaged audio data, and imaged tactile data. The videodata includes, for example, the left-eye video data and the tight-evevideo data. The audio data includes, for example, the audio data of theleft ear and the audio data of the right ear. The tactile data includes,for example, the fingertip data.

Each of the communication frames F1 to Fj includes information ofconsecutive serial numbers (frame numbers) in the order of transmission.The terminal apparatus 21-1 and the robot control apparatus 41-1 canspecify the timings of the respective communication frames F1 to Fj onthe basis of such serial numbers (frame numbers). It should be notedthat communication packets may be used instead of the communicationframes.

Even if some the communication frames F1 to Fj are lost in the secondsession from the robot control apparatus 41-1 to the terminal apparatus21-1, the control part 115 and the control part 215 can discard data orperform retransmission control in units of the communication framesincluding the data of all media (video, audio, and tactile) in responseto the elapse of a time-out period. The control part 115 and the controlpart 215 may retransmit the corresponding data if the delay time isequal to or less than the predetermined threshold value, and may discardthe corresponding data if the delay time is greater than thepredetermined threshold value.

[Motion Buffering]

The data transmitted and received between the terminal apparatus 21-1and the robot control apparatus 41-1 may be lost during datatransmission. In such cases, the control part 115 of the terminalapparatus 21-1 may interpolate the lost data using the data stored inthe buffer 112 so that the robot 31-1 operates as intended by the user11-1 and the operability of the user 11-1 improves. Thus, for example,even if some data is lost, the control part 115 of the terminalapparatus 21-1 can estimate and reproduce the lost data on the basis ofother data that has not been lost.

Similarly, the control part 215 of the robot control apparatus 41-1 cancompensate for any intermediate data by interpolating the data using thedata stored in the buffer 212. As a result, the control part 215 of therobot control apparatus 41-1 can estimate and reproduce the lost data onthe basis of other data that has not been lost even if some data hasbeen lost, for example. Such an interpolation may be applied, forexample, to any one or both of the data of the first session and thedata of the second session.

[Summary of the Configuration and Operation of the Robot System 1]

As described above, the robot system 1 according to the embodiment canrealize high-quality robot control by compensating for the delay byusing the buffer 112 and the buffer 212 in telexistence. For example,the robot system 1 temporarily stores the operator state information andthe robot state information in the buffer 112 and the buffer 212, sothat the variation amount of the delay time required from when thetransmission part 131 transmits the operator state information to whenthe reception part 132 receives the robot state information can besuppressed within the predetermined range. This makes it easier for theuser 11.-1 using the robot system 1 to operate the robot 31-1.

If the delay time required from when the transmission part 131 transmitsthe operator state information to when the reception part 132 receivesthe robot state information is longer than the predetermined timeperiod, the robot system 1 imparts the sensation based on the virtualstate information to the user 11-1. With such a configuration of therobot system 1, even if the delay time of data transmission in thenetwork 51 is long, it is possible to shorten the time from theoperation by the user 11-1 to the acquisition of the sensationcorresponding to the state of the robot 31-1. As a result, the user 11-1can comfortably control the robot 31-1.

It should be noted that the robot system 1 may control the delay time ineach of the buffer 112 on the user 11-1 side and the buffer 212 on therobot 31-1 side in another manner. As an example, the robot system 1 maybe controlled to hold bi-directional data (data of the first session anddata of the second session) for a predetermined period of time in eachof the buffer 112 on the user 11-1 side and the buffer 212 on the robot31-1 side, and output the data every predetermined period of time. Thepredetermined period of time is determined in accordance with, forexample, the medium having the longest delay time, and in this example,the predetermined period of time is determined in accordance with thedelay time of the video data transmitted from the robot controlapparatus 41-1 to the terminal apparatus 21-1.

Here, the number of users 11-1 to 11-n, the number of terminalapparatuses 21-1 to 21-n, the number of robot control apparatuses 41-1to 41-n, and the number of robots 31-1 to 31-n may be any number. Forexample, the number of users 11-1 to 11-n and the number of robots 31-1to 31-n may be the same or different.

In addition, in the embodiment, a configuration is shown in which theterminal apparatus 21-1 is provided with the buffer 112 and the controlpart 115 for controlling the delay time in the buffer 112 on the user11-1 side, but as another configuration, one or both of the buffer 112and the delay time control function in the buffer 112 may be provided ina control apparatus different from the robot control apparatus 41-1.

Similarly, in the embodiment, a configuration is shown in which therobot control apparatus 41-1 is provided with the buffer 212 and thecontrol part 215 for controlling the delay time in the buffer 212 on therobot 31-1 side, but the present invention is not limited to such aconfiguration. As another configuration, one or both of the buffer 212and the delay time control function in the buffer 212 may be provided ina control apparatus different from the robot control apparatus 41-1.

Here, in the robot system 1, one or more pieces of data of the visual,auditory, and tactile sensations and the like may be detected andtransmitted to the side of the robot 31-1 by the terminal apparatus 21-1on the side of the user 11-1, and the robot control apparatus 41-1 mayreceive the data on the side of the robot 31-1 to reproduce thesensation (such as, the visual, auditory, or tactile sensation) based onthe data. In this case, the terminal apparatus 21-1 includes, forexample, a sensor (for example, an image capturing device (a camera))for detecting a video image, a sensor (for example, a microphone) fordetecting sound, a sensor (for example, a pressure sensor) for detectingthe tactile sensation, or the like as the detection part 111.

Further, the robot control apparatus 41-1 includes, for example, asensation reproduction part that reproduces a sensation based on datasuch as the video, audio, and tactile data received from the terminalapparatus 21-1. The function of the sensation reproduction part may beprovided in, for example, the robot driving part 213, or may be providedseparately from the robot driving part 213. A part or all of thefunction of the sensation reproduction part may be mounted on the robot31-1. The sensation reproduction part in this case is, for example, avideo display device for reproducing a video image, a speaker forreproducing sound, a pressure generator (a motion generator) forreproducing the tactile sensation, or the like.

[Modification 1]

In the robot system 1, a plurality of users may control one sharedrobot. A case where three users 11-1 to 11-3 control one robot 31-1 isshown as an example. In this example, the terminal apparatuses 21-1 to21-3 of the three users 11-1 to 11-3 communicate with the robot controlapparatus 41-1 in a time-sharing manner to control the robot 31-1. Forexample, the robot 31-1 includes a plurality of movable parts, and thethree users 11-1 to 11-3 respectively control different movable parts.The plurality of movable parts is, for example, a plurality of arms.

FIG. 8 is a block diagram illustrating a schematic configuration oftime-sharing communication frames Z1 to Z6 according to the embodimentof the present invention. FIG. 8 shows a plurality of the consecutivecommunication frames Z1 to Z6. In this example, the terminal apparatus21-i of the first user 11-1 communicates with the robot controlapparatus 41-1 using the first communication frame Z1, for example. Theterminal apparatus 21-2 of the second user 11-2 communicates with therobot control apparatus 41-1 using the second communication frame Z2.The terminal apparatus 21-3 of the third user 11-3 communicates with therobot control apparatus 41-1 using the third communication frame Z3. Inthis manner, the plurality of terminal apparatuses 21-1 to 21-3communicate with the shared robot control apparatus 41-1 in atime-sharing manner, thereby controlling the shared robot 31-1.

[Modification 2]

In the robot system 1, one user may control a plurality of robots. Acase where the one user 11-1 controls three robots 31-1 to 31-3 is shownas an example. In this embodiment, the terminal apparatus 21-1 of theone user 11-1 controls the three robots 31-1 to 31-3 by communicatingwith three robot control apparatuses 41-1 to 41-3 in a time-sharingmanner.

The description will be made using the example of FIG. 8. In thisexample, the terminal apparatus 21-1 of the one user 11-1 communicateswith the robot control apparatus 41-1 of the first robot 311 using thefirst communication frame Z1. The terminal apparatus 21-1 communicateswith the robot control apparatus 41-2 of the second robot 31-2 using thesecond communication frame Z2 The terminal apparatus 21-1 communicateswith the robot control apparatus 41-3 of the third robot 31-3 using thethird communication frame Z3. In this manner, the one terminal apparatus21-1 communicates with the plurality of robot control apparatuses 41-1to 41-3 in a time-sharing manner, thereby controlling the plurality ofrobots 31-1 to 31-3.

[Modification 3]

The robot system 1 may further include a management apparatus connectedto the terminal apparatuses 21-1 to 21-n and the robot controlapparatuses 41-1 to 41-n via the network 51. The management apparatusis, for example, a server.

A management apparatus 61 is connected to the network 51, and relays thecommunication between the terminal apparatuses 21-1 to 21-n and therobot control apparatuses 41-1 to 41-n. That is, the managementapparatus 61 receives a signal transmitted from the terminal apparatuses21-1 to 21-n to the network 51 and transmits the signal to the robotcontrol apparatus 41-1 to 41-n of the destination, and further, themanagement apparatus 61 receives a signal transmitted from the robotcontrol apparatus 41-1 to 41-n to the network 51 and transmits thesignals to the terminal apparatuses 21-1 to 21-n of the destination.

As described above, the robot system may include the managementapparatus 61 on the cloud to manage the communication performed betweenthe terminal apparatuses 21-1 to 21-n and the robot control apparatuses41-1 to 41-n. For example, the management apparatus 61 storesinformation of each address of the terminal apparatuses 21-1 to 21-n andthe robot control apparatuses 41-1 to 41-n corresponding to each other,and relays the signals communicated between them on the basis of theinformation of the addresses. The management apparatus 61 may have amachine learning function, and may correct, on the basis of a result ofthe machine learning, an operation in which the users 11-1 to 11-ncontrol the robots 31-1 to 31-n.

In the embodiment described while referencing FIGS. 2 to 6, the terminalapparatus 21-1 includes the buffer 112 and the function of adjusting thedelay time of the data in the buffer 112. Further, the robot controlapparatus 41-1 includes the buffer 212 and the function of adjusting thedelay time of the data in the buffer 212. Any one or both of these twofunctions may be provided in the management apparatus 61.

As an example, the management apparatus 61 may include the buffer 112and the function of adjusting the delay time of data in the buffer 112(referred to as a function C1 in this example for convenience ofdescription) instead of the terminal apparatus 21-1. In this case, theterminal apparatus 21-1 does not need to have such a function C1.

As another example, the management apparatus 61 may include the buffer212 and the function of adjusting the delay time of the data in thebuffer 212 (referred to as a function C2 in this example for convenienceof description) instead of the robot control apparatus 41-1. In thiscase, the robot control apparatus 41-1 does not need to have such afunction C2.

Further, as another example, the management apparatus 61 may includeboth the function C1 and the function C2. In particular, if thecommunication delay is small, it is considered preferable that themanagement apparatus 61 has the function C1 or the function C2. Forexample, if the delay of the data communication from the terminalapparatus 21-1 to the robot control apparatus 41-1 via the network 51 issmaller than the delay of the data communication from the robot controlapparatus 41-1 to the terminal apparatus 21-1 via the network 51, theconfiguration in which the terminal apparatus 21-i includes the functionC1 and the management apparatus 61 includes the function C2 may be used.

FIG. 9 is a diagram illustrating the management apparatus 61 capable ofcommunicating with the terminal apparatus 21-1 and the robot controlapparatus 41-1 via the network 51. The management apparatus 61 includesa delay specification part 611 and a notification part 612. The delayspecification part 611 specifies the delay time from when the terminalapparatus 21-1 transmits the operator state information to when theterminal apparatus 21-1 receives the robot state information, on thebasis of the time at which the operator state information is receivedfrom the terminal apparatus 21-1 and the time at which the robot stateinformation is received from the robot control apparatus 41-i. Thenotification part 612 notifies the control part 115 of the terminalapparatus 21-1 about the delay time specified by the delay specificationpart 611.

The control part 115 may decide to impart the sensation based on thevirtual state information to the user 11-1 on the basis of aninstruction from the management apparatus. In this case, the delayspecification part 611 of the management apparatus specifies the factorcausing the delay, and the notification part 612 instructs the controlpart 115 of the terminal apparatus 21-1 to impart the sensation based onthe virtual state information to the user 11-1 if the factor specifiedby the delay specification part 611 is the transmission delay of thenetwork 51. On the basis of the instruction from the notification part612, the control part 115 controls the sensation imparting part 114 toimpart the sensation based on the virtual state information to the user11-1. In this manner, the control part 115 imparts the sensation basedon the virtual state information on the basis of the instruction fromthe management apparatus, and therefore the control part 115 can impartthe sensation based on whichever information, either the virtual stateinformation or the robot state information, is more suitable for thedelay state of the networks 51.

The notification part 612 may instruct the robot control apparatus 41-1to transmit the virtual state information to the terminal apparatus 21-1if the factor specified by the delay specification part 611 is theoperation delay of the robot. FIG. 10 is a diagram illustrating therobot system 1 in which the robot control apparatus 41-1 provides thevirtual state information to the user 11-1. The robot control apparatus41-1 provides the terminal apparatus 21-1 with, for example, the virtualstate information selected from the plurality of pieces of the virtualstate information stored in the storage part 251. With such aconfiguration of the robot system 1, even if the terminal apparatus 21-1does not store the virtual state information, the terminal apparatus21-1 can impart the sensation based on the virtual state information tothe user 11-1.

Summary of the Embodiment

In the robot system 1, the terminal apparatus (in the example of FIG. 1,the terminal apparatuses 21-1 to 21-n) transmits the data relating tothe user (in the example of FIG. 1, the users 11-1 to 11-n) to the robotcontrol apparatus (in the example of FIG. 1, the robot controlapparatuses 41-1 to 41-n) via the network (in the example of FIG. 1, thenetwork 51). The robot control apparatus controls the robot (in FIG. 1,robots 31-1 to 31-n) on the basis of the data.

In addition, the robot control apparatus transmits the data relating tothe robot to the terminal apparatus via the network, and the terminalapparatus imparts the sensation to the user on the basis of the data. Atleast one of the terminal apparatus and the robot control apparatusincludes (i) the buffer (the buffers 112 and 212 in the examples ofFIGS. 2 and 3) for storing at least one piece of data from among thedata to be transmitted and the data received, and (ii) the control part(the control parts 115 and 215 in the examples of FIGS. 2 and 3) forcontrolling the delay time of the data stored in the buffer. Since therobot system 1 has such a configuration, the robot system 1 can adjustthe transmission delay of data so as not to cause trouble in theoperation of the robot by the user thereby improving the operability ofthe robot in telexistence.

In one configuration, when transmitting the data to be transmitted andthe received data, the control part controls the delay time of apredetermined transmission between the network and the user or betweenthe network and the robot to be within the predetermined range.

In one configuration, the data related to the user includes the datarelated to the motion of the user, the data related to the robotincludes the video data, the audio data, and the tactile data in therobot, and the robot control apparatus communicates the video data, theaudio data, and the tactile data in a shared communication frame. In oneconfiguration, the data is interpolated using the data stored in thebuffer for at least one of the terminal apparatus and the robot controlapparatus.

In one configuration, the terminal apparatus imparts the sensation basedon the robot state information to the user if the delay time requiredfrom transmitting the operator state information to receiving the robotstate information is equal to or less than the predetermined timeperiod, and imparts the sensation based on the virtual state informationindicating the estimated state of the robot to the user if the delaytime is longer than the predetermined time period. Because the terminalapparatus works in this manner, even if the delay time is long, the usercan quickly feel the sensation corresponding to the state of the robot,thereby improving the operability.

A program for realizing the functions of the respective apparatuses (forexample, the terminal apparatuses 21-1 to 21-n or the robot controlapparatus 41-1 to 41-n) according to the embodiment described above maybe recorded in a computer-readable recording medium (a storage medium),and the program recorded in the recording medium may be read into acomputer system and executed to perform the process.

The “computer system” herein may include an operating system (OS) orhardware such as a peripheral device. The “computer-readable recordingmedium” refers to a writable non-volatile memory such as a flexibledisk, a magneto-optical disk, a ROM, a flash memory, a portable mediumsuch as a digital versatile disc (DVD), or a storage device such as ahard disk incorporated in a computer system. It should be noted that therecording medium may be, for example, a recording medium detachable froma computer. Further, the recording medium may be, for example, arecording medium that temporarily records data.

Furthermore, the “computer-readable recording medium” includes a mediumthat holds a program for a predetermined period of time, such as avolatile memory (for example, a dynamic random access memory (DRAM)) ina computer system serving as a server or a client when the program istransmitted via a network such as the Internet or a communication linesuch as a telephone line.

The program may be transmitted from a computer system storing theprogram in a storage device or the like to another computer system via atransmission medium or by a transmission wave in the transmissionmedium. Here, the “transmission medium” for transmitting a programrefers to a medium having a function of transmitting information, forexample, a network (communication network) such as the Internet or atelecommunication line (communication line) such as a telephone line.

Further, the above-mentioned program may be a program for realizing apart of the above-mentioned functions. Furthermore, the above-mentionedprogram may be a so-called difference file (difference program) capableof realizing the above-mentioned functions in combination with a programalready recorded in the computer system.

The embodiment of the present invention is explained above in detailwhile referencing the drawings. The specific configuration is notlimited to above embodiment and it is possible to make variousmodifications within the scope of the invention.

The present invention has been described above on the basis of theexemplary embodiments. The technical scope of the present invention isnot limited to the scope explained in the above embodiments, and it isobvious to those skilled in the art that various changes andmodifications within the scope of the invention may be made. An aspectto which such changes and modifications are added can be included in thetechnical scope of the present invention is obvious from the descriptionof the claims.

What is claimed is:
 1. A sensation imparting apparatus comprising: atransmission part that transmits operator state information indicating astate of an operator operating a robot to the robot; a reception partthat receives robot state information indicating a state of the robotfrom the robot; a sensation imparting part that imparts a predeterminedsensation to the operator; and a control part that controls thesensation imparting part to impart a sensation based on the robot stateinformation to the operator if a delay time required from when thetransmission part transmits the operator state information to when thereception part receives the robot state information is equal to orshorter than a predetermined time period, and controls the sensationimparting part to impart a sensation based on virtual state informationindicating an estimated state of the robot to the operator if the delaytime is longer than the predetermined time period.
 2. The sensationimparting apparatus according to claim 1, wherein the control partcontrols the sensation imparting part to impart the sensation based onthe robot state information received by the reception part to theoperator after imparting the sensation based on the virtual stateinformation to the operator.
 3. The sensation imparting apparatusaccording to claim 2, wherein the control part controls the sensationimparting part to impart, to the operator, a sensation interpolatedbetween a first sensation based on the virtual state information and asecond sensation based on the robot state information received by thereception part after imparting the sensation based on the virtual stateinformation to the operator, if there is a difference of a predeterminedmagnitude or more between the first sensation and the second sensation.4. The sensation imparting apparatus according to claim 1, furthercomprising: a buffer part that temporarily stores one or more pieces ofthe robot state information received by the reception part and thensequentially outputs the one or more pieces of the robot stateinformation to the sensation imparting part, wherein the control partcontrols the sensation imparting part to impart a sensation based onvirtual state information to the operator if an amount of the one ormore pieces of the robot state information stored in the buffer partbecomes less than a threshold value.
 5. The sensation impartingapparatus according to claim 1, further comprising: a buffer part thattemporarily stores one or more pieces of the robot state informationreceived by the reception part and then sequentially outputs the one ormore pieces of the robot state information to the sensation impartingpart, wherein the control part controls an amount of the robot stateinformation temporarily stored in the buffer part so that a variationamount of a delay time required from when the transmission parttransmits the operator state information to when the reception partreceives the robot state information falls within a predetermined range.6. The sensation imparting apparatus according to claim 1, wherein thecontrol part includes a storage part that stores a plurality of piecesof the virtual state information corresponding to the robots indifferent states, and controls the sensation imparting part to impart asensation based on virtual state information selected from the pluralityof pieces of the virtual state information to the operator on the basisof the operator state information.
 7. The sensation imparting apparatusaccording to claim 6, wherein the storage part stores the plurality ofpieces of the virtual state information including virtual imagescorresponding to images captured by the robots in different states, andthe control part controls the sensation imparting part to impart, to theoperator, a sensation based on virtual state information including thevirtual image corresponding to a position of the robot specified on thebasis of an operation content indicated by the operator stateinformation among the plurality of pieces of the virtual stateinformation.
 8. The sensation imparting apparatus according to claim 6,wherein the control part estimates a state of the robot after theoperator operates, on the basis of the state of the operator indicatedby the immediately preceding operator state information and the state ofthe operator indicated by the most recent operator state information, inorder to select one piece of virtual state information from theplurality of pieces of the virtual state information, and selects onepiece of virtual state information corresponding to the estimated stateof the robot.
 9. The sensation imparting apparatus according to claim 6,wherein the control part stores sensation hist information indicatingcontent of the sensation based on the robot state information impartedby the sensation imparting part in the storage part, and selects virtualstate information to be used for imparting a sensation to the operatorfrom the plurality of pieces of the virtual state information on thebasis of the content of the sensation indicated by the sensation historyinformation stored immediately before.
 10. The sensation impartingapparatus according to claim 7, wherein the control part specifies afactor causing a delay from when the transmission part transmits theoperator state information to when the reception part receives the robotstate information, and determines, on the basis of the specified factor,whether to control the sensation imparting part to impart the sensationbased on the virtual state information stored in the storage part to theoperator or to control the sensation imparting part to impart thesensation based on the virtual state information received via a networkto the operator.
 11. A robot control system comprising: a sensationimparting apparatus that transmits, to a network, operator stateinformation indicating a state of an operator operating a robot; a robotcontrol apparatus that controls the robot on the basis of the operatorstate information received via the network; and a management apparatusthat is capable of communicating with the robot control apparatus andthe sensation imparting apparatus via the network, wherein the sensationimparting apparatus includes: a transmission part that transmits theoperator state information indicating the state of the operatoroperating the robot to the robot; a reception part that receives robotstate information indicating a state of the robot from the robot; asensation imparting part that imparts a predetermined sensation to theoperator, and a control part that controls the sensation imparting partto impart a sensation based on the robot state information to theoperator if a delay time required from when the transmission parttransmits the operator state information to when the reception partreceives the robot state information is equal to or shorter than apredetermined time period, and controls the sensation imparting part toimpart a sensation based on virtual state information indicating anestimated state of the robot to the operator if the delay time is longerthan the predetermined time period, wherein the management apparatusincludes: a delay specification part that specifies the delay time; anda notification part that notifies the control part about the delay timespecified by the delay specification part.
 12. The robot control systemaccording to claim 11, wherein the delay specification part specifies afactor causing the delay time, the notification part instructs thesensation imparting apparatus to impart the sensation based on thevirtual state information to the operator when the factor specified bythe delay specification part is a transmission delay of the network, andthe control part of the sensation imparting apparatus controls thesensation imparting part to impart the sensation based on the virtualstate information to the operator.
 13. The robot control systemaccording to claim 12, wherein the notification part instructs the robotcontrol apparatus to transmit virtual state information to the sensationimparting apparatus if the factor specified by the delay specificationpart is an operation delay of the robot, and the control part of thesensation imparting apparatus controls the sensation imparting part toimpart the sensation based on the virtual state information receivedfrom the robot control apparatus to the operator.
 14. A robot controlmethod, the method comprising the computer-implemented steps of:transmitting operator state information indicating a state of anoperator operating a robot to the robot; determining whether or not adelay time required from transmitting the operator state information toreceiving robot state information indicating a state of the robot isequal to or less than a predetermined time period; and imparting asensation based on the robot state information to the operator if thedelay time is equal to or less than the predetermined time period, andimparting a sensation based on virtual state information indicating anestimated state of the robot to the operator if the delay time is longerthan the predetermined time period.